Anomaly-based Intrusion Detection from Traffic Datamining on Internet Connections*

In this paper, we present a new datamining approach to generating frequent episode rules for the construction of anomaly-based, intrusion detection systems (IDS). These rules are derived from normal network traffic profiles. An anomaly is detected when the rule deviates significantly from the normal patterns. Three rule pruning techniques are devised to reduce the rule search space by 50-80%. This reduction makes datamining viable in detecting unknown network attacks. The new approach accelerates the entire process of machine learning and profile matching for intrusion detection. Testing our new scheme over DARPA 1999 IDS evaluation data sets, we find a 13% reduction in false alarms over 50 network attack incidents. The network episode rules reveal inter-relationship among sequences of network connection events. We detect unknown attacks embedded in telnet, http, ftp, smtp, and other requests of TCP, UDP or ICMP connections. Our IDS leads to an intrusion detection rate up to 47% for DoS (denial of service), R2L (remote-to-local), and probe attacks. Our scheme detects many attacks that cannot be detected by Snort, including the smurf, Apache2, Guesstelnet, Dict, Neptune, and Udpstorm. We recommend the use of the proposed anomaly detection scheme jointly with signature-based IDS to yield even better results. These results prove the viability of using the new scheme to build automated intrusion detection and response systems in real time.